A Principled Framework for Evaluating Summarizers: Comparing Models of Summary Quality against Human Judgments
نویسندگان
چکیده
We present a new framework for evaluating extractive summarizers, which is based on a principled representation as optimization problem. We prove that every extractive summarizer can be decomposed into an objective function and an optimization technique. We perform a comparative analysis and evaluation of several objective functions embedded in wellknown summarizers regarding their correlation with human judgments. Our comparison of these correlations across two datasets yields surprising insights into the role and performance of objective functions in the different summarizers.
منابع مشابه
Sentiment Summarization: Evaluating and Learning User Preferences
We present the results of a large-scale, end-to-end human evaluation of various sentiment summarization models. The evaluation shows that users have a strong preference for summarizers that model sentiment over non-sentiment baselines, but have no broad overall preference between any of the sentiment-based models. However, an analysis of the human judgments suggests that there are identifiable ...
متن کاملBayesian Learning in Text Summarization
The paper presents a Bayesian model for text summarization, which explicitly encodes and exploits information on how human judgments are distributed over the text. Comparison is made against non Bayesian summarizers, using test data from Japanese news texts. It is found that the Bayesian approach generally leverages performance of a summarizer, at times giving it a significant lead over nonBaye...
متن کاملارائه یک سیستم هوشمند و معناگرا برای ارزیابی سیستم های خلاصه ساز متون
Nowadays summarizers and machine translators have attracted much attention to themselves, and many activities on making such tools have been done around the world. For Farsi like the other languages there have been efforts in this field. So evaluating such tools has a great importance. Human evaluations of machine summarization are extensive but expensive. Human evaluations can take months to f...
متن کاملFuzzy Multi-criteria decision making approach for human capital evaluation of municipal districts
People in every organization could be considered as the most important resource which contributes to the development of that organization. In fact, human capital is the most important dimension of organization’s intellectual capital especially in service-oriented organizations like municipality. Therefore, the main purpose of this paper is to introduce a suitable framework for human capital eva...
متن کاملA More Cohesive Summarizer
We have developed a cohesive extraction based single document summarizer (COHSUM) based on coreference links in a document. The sentences providing the most references to other sentences and that other sentences are referring to, are considered the most important and are therefore extracted. Additionally, before evaluations of summary quality, a corpus analysis was performed on the original doc...
متن کامل